A study of first-passage time minimization via Q-learning in heated gridworlds

Optimization of first-passage times is required in applications ranging from nanobots navigation to market trading. In such settings, one often encounters unevenly distributed noise levels across the environment. We extensively study how a learning agent fares 1- and 2- dimensional heated gridworlds with an uneven temperature distribution. The results show certain bias effects agents trained via simple tabular Q-learning, SARSA, Expected SARSA Double Q-learning. While high rate prevents exploration regions higher temperature, low enough increases presence regions. discovered peculiarities biases temporal-difference-based reinforcement methods should be taken into account real-world physical design.